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FOREWORD 
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The  Selection  &  Classification  Technical  Area  of  the  Army 
Research  Institute  for  the  Behavioral  A  Social  Sciences  (ARI)  is 
concerned  with  developing  effective  procedures  for  the  selection  of 
applicants  into  military  service  and  for  the  classification  of 
accessions  into  Array  occupational  specialities.  The  purpose  of  this 
research  was  to  examine  and  document  the  construct  validity  of  the 
verbal  subtests  of  the  current  Department  of  Defense  military  selec¬ 
tion  and  classification  battery,  the  Armed  Services  Vocational 
Aptitude  Battery  ( ASVAB)  8/9/10. 


EDGAR  M.  JOHNSON 
Technical  Director 


COGNITIVE  PROCESSING  DETERMINANTS  OF  ITEM  DIFFICULTY 
ON  THE  VERBAL  SUBTESTS  OF  THE  ARMED  SERVICES 
VOCATIONAL  APTITUDE  BATTERY 


BRIEF 


Requirement : 

To  identify  the  cognitive  processing  operations,  stores,  and 
strategies  involved  in  performance  on  the  verbal  subtests  of  the 
Armed  Services  Vocational  Aptitude  Battery  ( ASVAB)  8/9/10. 


Procedure: 

Cognitive  dimensions  thought  to  underlie  performance  on  the 
ASVAB  Word  Knowledge  (WK)  and  Paragraph  Comprehension  (PC)  items 
were  operationlized;  the  verbal  items  on  two  ASVAB  forms  were  rated 
on  these  variables.  The  relative  effects  of  the  cognitive  dimen¬ 
sions  on  Rasch  item  difficulties  were  assessed  for  eight  groups  of 
FY81  Army  applicants  and  accessions  using  Linear  Logistic  Latent 
Trait  methods. 


Findings : 

Analyses  suggested  that  cognitive  processing  variables  were 
related  to  item  difficulty  for  FY81  applicants  and  accessions  on  the 
ASVAB  8/9/10  verbal  subtests.  The  LLLT  models  predicted  from  17%  to 
30%  of  the  variance  in  the  item  difficulty  values  estimated  by  the 
Rasch  models  for  the  WK  items.  The  results  for  the  PC  items  were 
inconclusive. 


Utilization  of  Findings: 

The  relevance  of  cognitive  processing  variables  to  item  dif¬ 
ficulty  for  the  ASVAB  8/9/10  WK  and  PC  subtests  was  demonstrated. 
Further  examination  of  the  predictive  utility  of  these  and  similar 
constructs  to  success  in  Army  training  and  military  jobs  is  needed. 
Analyses  of  this  type  will  enable  assessment  and  documentation  of  the 
construct  validity  of  ASVAB  subtests.  These  methods  can  be  used  to 
develop  item  sets  so  that  specified  training-  and  job-relevant 
processing  abilities  are  tapped. 
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COGNITIVE  PROCESSING  DETERMINANTS  OF  VERBAL 
ITEM  DIFFICULTY  ON  THE  ARMED  SERVICES 
VOCATIONAL  APTITUDE  BATTERY 


ABSTRACT 

Research  was  designed  to  identify  cognitive  processing 
operations  and  strategies  involved  in  performance  on  the 
verbal  subtests  of  the  Armed  Services  Vocational  Aptitude 
Battery  (ASVAB) .  Cognitive  processes  thought  to  underlie 
performance  on  the  Word  Knowledge  and  Paragraph  Comprehen¬ 
sion  subtests  were  identified.  Items  on  two  ASVAB  forms 
were  rated  on  these  variables.  The  relative  effects  of 
cognitive  processes  were  examined  for  eight  groups  of  A> — 
Applicants  and  Accessions  using  Linear  Latent  Logistic 
Trait  methods.  Analyses  demonstrated  the  relevance  < 
cognitive  processing  variables  to  item  difficulty  fo  .he 
ASVAB  verbal  items. 


INTRODUCTION 

The  purpose  of  this  research  was  to  identify  the  cogni¬ 
tive  processing  operations,  stores,  and  strategies  involved 
in  performance  on  the  verbal  subtests  of  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  8/9/10.  Procedures 
were  developed  for  evaluating  the  underlying  cognitive 
ability  contributions  to  performance  on  item  sets. 

The  methods  were  proposed  for  general  use  in  assessing  and 
documenting  cognitive  processing  contributions  to  perfor¬ 
mance  on  tests  and  test-like  tasks.  The  procedures  that 
were  developed  were  proposed  for  use  in  constructing 
industrial/organizational  assessment  instruments  that  tap 
specified,  training-  and  job-relevant  processing  abilities. 

The  specific  objective  of  this  work  was  to  clarify 
the  construct  validity  of  verbal  items  on  ASVAB  8/9/10. 
These  results  were  used  in  subsequent  analyses  of  the 
predictive  validity  of  assessed  ability  components  to 
successful  performance  in  Army  jobs.  The  research  was 
conducted  in  two  parts. 


METHOD 


Part  1  began  with  a  re  riew  of  recent  work  in  the  area  of 
verbal  information  processing.  The  review  was  directed  at  iden¬ 
tifying  possible  sources  of  cognitive  processing  operation,  store, 
and  strategy  differences  on  the  verbal  items  of  ASVAB  and  suggest¬ 
ing  methodological  alternatives  for  the  examination  of  relevant 
data.  Part  1  activities  included: 

A.  Conducting  a  review  of  the  definition  and  measurement  of 
cognitive  processing  components  involved  in  performance  on  verbal 
tasks . 


B.  Developing  a  model  of  verbal  performance  to  provide  a 
conceptual  framework  for  identifying  and  operationalizing  process¬ 
ing  components. 

C.  Drafting  theoretical  and  operational  definitions  for  the 
ability  components  deemed  relevant  to  performance  on  the  Word 
Knowledge  (WK)  and  Paragraph  Comprehension  (PC)  items  of  ASVAB. 

D.  Coding  the  Word  Knowledge  and  Paragraph  Comprehension  items 
on  two  ASVAB  8/9/10  forms  for  the  presence  or  absence,  complexity, 
or  frequency  of  execution  for  each  cognitive  ability  component. 

Part  1  methods  were  based  on  the  premise  that  variation  in 
item  difficulty  would  be  a  function  of  the  type  and  level  of 
required  processing.  It  was  also  posited  that  processing  require¬ 
ments  could  be  accounted  for  by  structural  features  of  items  and 
that  processing  requirements  could  be  indexed  or  quantified 
accord ingly . 

Part  2  analyses  examined  the  ability  component  contributions 
to  item  and  person  variance  on  the  verbal  items  of  ASVAB.  Part  2 
activities  included: 

A.  Modeling  item  difficulty  for  the  Word  Knowledge  and 
Paragraph  Comprehension  items  for  Fiscal  Year  1981  Army  Applicants 
and  Accessions  using  linear  logistic  latent  trait  techniques; 
stimulus  complexity  data  reflecting  the  nature  or  level  of  in¬ 
volvement  for  relevant  cognitive  processing  operations  and  per¬ 
formance  data  for  selected  samples  of  examinees  fed  into  the 
models. 


B.  Cross  validating  the  models  across  ASVAB  forms  and  samples 
of  examinees. 

A  description  of  the  operational  Armed  Services  Vocational  Ap¬ 
titude  Battery  (ASVAB)  follows. 


The  Armed  Services  Vocational  Aptitude  Battery. 

The  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  was 
introduced  in  January  1976  as  the  single  Department  of  Defense 
selection  and  classification  battery.  Six  replacement  forms  were 
developed  and  implemented  October  1,  1980.  The  1980  version, 

ASVAB  8/9/10,  was  the  focus  of  this  work. 

The  ASVAB  is  administered  to  over  one  million  applicants  for 
military  service  each  year.  The  ASVAB  plays  two  important  roles 
in  the  enlistment  of  military  personnel.  First,  ASVAB  scores  are 
used  in  determining  an  applicant's  eligibility  for  military 
service.  Second,  ASVAB  scores  are  used  to  establish  an 
individual's  eligibility  for  assignment  to  specific  military  jobs. 

The  ASVAB  consists  of  eight  power  and  two  speed  tests.  The 
ten  subtests  are  listed  in  Table  1 .  These  subtests  are  included 
because  research  and  experience  have  demonstrated  that  they  are 
valid  predictors  of  success  in  various  types  of  military  jobs. 

Table  1  also  provides  the  number  of  items  per  subtest  and  the 
testing  time  limits.  Although  all  ten  subtests  are  individually 
timed,  only  Numerical  Operations  and  Coding  Speed  are  considered 
speed  tests.  The  remaining  eight  subtests  are  power  tests  with 
ample  time  limits. 

The  means  and  standard  deviations  included  in  Table  1  were 
computed  for  the  subtest  raw  scores  of  a  random  sample  of  FY81 
Army  Applicants  described  in  Kass,  Mitchell,  Grafton,  and  Wing 
(1981).  Estimates  of  subtest  internal  consistency  reliabilities 
for  the  eight  power  subtests  for  each  of  the  six  forms  of  ASVAB 
8/9/10  were  derived  from  Ree,  Mullins,  Mathews  and  Massey  (1981). 
Parallel  form  reliability  estimates  for  the  two  speeded  subtests 
were  obtained  from  Sims  and  Hiatt  (1981)  and  Wilfong  (1980). 

The  Word  Knowledge  Subtest 

There  are  thirty-five  items  on  the  Word  Knowledge  subtest  of 
the  operational  ASVAB.  The  test  is  constructed  to  assess  the 
examinee's  understanding  of  the  meaning  of  words.  Surface  charac¬ 
teristic  differences  associated  with  stem  and  alternatives  for  the 
vocabulary  items  include:  word  frequency,  vocabulary  level, 
number  of  syllables,  grammatical  class,  stem  type,  etc.  Instruc¬ 
tions  and  sample  items  from  the  Word  Knowledge  subtest  appear  at 
Table  2. 


The  Paragraph  Comprehension  Subtest 

The  ASVAB  8/9/10  Paragraph  Comprehension  subtest  has  fifteen 
items.  The  subtest  is  constructed  to  assess  the  examinee's 
understanding  of  the  meaning  of  paragraphs.  Surface  characteris- 


3 


TABLE  1  (continued) 


Table  2 

The  Word  Knowledge  Subtest 


This  test  has  questions  about  the  meaning  of  words.  Each 
question  has  an  underlined  word.  You  are  to  decide  which  of 
the  four  possible  answers  most  nearly  means  the  same  as  the 
underlined  word,  then  blacken  the  appropriate  space  on  your 
answer  sheet. 

1.  Snail  most  nearly  means 

A.  sturdy 

B.  round 

C.  cheap 

D.  little 

2.  The  accountant  discovered  an  error. 

A.  searched 

B.  found 

C.  enlarged 

D.  entered 

3.  Inform  most  nearly  means 

A.  ask 

B.  turn 

C .  tell 

D.  ignore 

4.  The  wind  is  variable  today 

A.  shifting 

B.  chilling 

C.  steady 

D.  mild 

5*  Cease  most  nearly  means 

A.  start 

B.  change 

C.  continue 

D.  stop 


Table  3 

The  Paragraph  Comprehension  Subtest 


This  is  a  test  of  your  ability  to  understand  what  you  read. 
In  this  section  you  will  find  one  or  more  paragraphs  of 
reading  material  followed  by  incomplete  statements  or 
questions.  You  are  to  read  the  paragraph  and  select  one  of 
four  lettered  choices  which  best  completes  the  statement  or 
answers  the  question.  When  you  have  selected  the  answer, 
blacken  in  the  correct  nunbered  letter  on  your  answer 
sheet. 

1.  Prom  a  building  designer's  standpoint,  three  things  that 
make  a  home  livable  are  the  client,  the  building  site,  and 
the  amount  of  money  the  client  has  to  spend. 

According  to  the  passage,  to  make  a  home  livable 

A.  the  prospective  piece  of  land  makes  little  difference 

B.  it  can  be  built  on  any  piece  of  land 

C.  the  design  must  fit  the  owner's  income  and  site 

D.  the  design  must  fit  the  designer's  income 

2.  In  certain  areas  water  is  so  scarce  that  every  attest  is 
made  to  conserve  it.  Fbr  instance,  on  one  oasis  in  the 
Sahara  Desert  the  amount  of  water  necessary  for  each  date 
palm  tree  has  been  carefully  determined. 

How  much  water  is  each  tree  given? 

A.  no  water  at  all 

B.  exactly  the  amount  required 

C.  water  only  if  it  is  healthy 

D.  water  on  alternate  days 

3.  The  duty  of  the  lighthouse  keeper  is  to  keep  the  light 
burning  no  matter  what  happens,  so  that  ships  will  be  warned 
of  the  presence  of  dangerous  rocks.  If  a  shipwreck  should 
occur  near  the  lighthouse,  even  though  he  would  like  to  aid 
in  the  rescue  of  its  crew  and  passengers,  the  lighthouse 
keeper  must 

A.  stay  at  his  light 

B.  rush  to  their  aid 

C.  turn  out  the  light 

D.  quickly  sound  the  siren 


tic  differences  associated  with  the  passage,  question  stem,  and 
alternatives  include:  total  number  of  words,  mean  grade  level  of 
words,  standard  deviation  of  vocabulary  level,  number  of  polysyl¬ 
labic  words,  location  of  response-relevant  information  in  the 
passage,  type  of  passage  (factual,  fictional,  first  person,  second 
person,  etc.)  average  length  of  sentence,  number  of  sentences, 
number  of  associated  questions  and  type  of  question.  The 
paragraph  selections  on  the  PC  subtests  have  from  one  to  four 
accompanying  questions.  The  number  of  distinct  paragraph  selec¬ 
tions  for  the  six  operational  ASVAB  forms  ranges  from  six  to  nine. 
Instructions  and  sample  items  from  the  Paragraph  Comprehension 
subtest  are  at  Table  3* 

The  unit  of  analysis  for  the  Part  2  computations  was  the 
item.  The  two  ASVAB  forms  that  were  examined  in  this  work  each 
had  nine  independent  PC  items.  The  probability  that  the  ratio  of 
number  of  components  to  items  would  be  high  for  the  PC  subtests 
was  recognized.  The  decision  to  proceed  with  the  PC  analyses  in 
spite  of  this  was  based  on  the  fact  that  the  PC  items  look  par¬ 
ticularly  non-parallel  across  forms.  The  results  that  follow  for 
the  PC  items  were  merely  suggestive. 

Part  One 

The  Development  of  a  Model  of  Verbal  Performance 

The  model  of  verbal  performance  that  was  developed  for  this 
work  borrows  heavily  from  the  paradigms  of  Hunt,  Carroll, 
Prederiksen,  Pellegrino,  Kintsch,  and  others.  The  model  describes 
verbal  performance  in  the  context  of  text  analysis.  The  Services 
use  the  verbal  subtests  of  the  ASVAB  to  predict  general  verbal 
ability;  the  verbal  items,  however,  directly  assess  only  those 
skills  used  in  text  processing.  The  model,  therefore,  charac¬ 
terizes  verbal  performance  with  respect  to  the  processes  which 
underlie  text  comprehension.  The  model  does  not  define  a  general 
theory  of  cognitive  processing. 

The  model  depicts  verbal  performance  by  five  processing  or 
storage  structures.  The  first  structure  might  be  thought  of  as  a 
perceptual  processor,  the  second  as  an  executive  or  control 
processor,  the  third  as  the  locus  of  lexical  access  and  semantic- 
syntactic  analysis,  the  fourth  as  knowledge-based  information  and 
information-free  storage,  and  the  fifth  as  a  response 
processor.  Each  structure  is  discussed  below.  The  structures  are 
not  strictly  serially  or  hierarchically  ordered.  The  flow  of 
information  within  the  system  is  not  necessarily  sequential  or 
parallel.  A  schematic  of  the  model  is  at  Table  4. 

The  PERCEPTUAL  PROCESSOR  is  the  structure  that  inputs 
stimulus  information  to  the  processing  system.  It  includes  the 
set  of  operations  that  converts  the  physical  stimulxxs  to  a  form 
that  is  appropriate  for  the  task;  it  includes  the  operations  that 


Table  A 


Model  of  Verbal  Performance 
VISUAL  DISPLAY 


Perceptual  Processor 

A.  Visual  Feature  Extraction 

B.  Perceptual  Encoding 

C.  Decoding 


si/ 


Response 


match  stimuli  to  appropriate  grapheme  and  phoneme 
representations.  The  perceptual  processor  is  characterized  by 
three  operations  described  by  Frederiksen  as:  visual  feature 
extraction,  perceptual  encoding,  and  decoding.  Visual  feature 
extraction  is  the  operation  by  which  different  types  of 
information  about  the  stimulus  display  are  extracted.  The 
processor  may  select  and  organize  information  for  further 
processing  or  it  may  ignore  or  deeraphasize  stimuli.  Perceptual 
encoding  is  the  operation  by  which  information  is  input  into  the 
system;  external  stimuli  are  translated  to  internal  codes. 
Decoding  is  the  operation  by  which  arbitrary  physical  patterns 
are  recognized  as  representations  of  grapheme  and  phoneme 
concepts  in  the  lexicon.  These  operations  may  be  thought -of  as 
automated  processes  for  the  samples  of  examinees  considered  here. 
The  processor  may  be  thought  of  as  a  short  term  sensory  storage 
or  buffer  component. 

The  EXECUTIVE  OR  CONTROL  PROCESSOR  is  the  structure  that 
controls  the  flow  of  information  in  the  system  and  has  access  to 
the  various  levels  of  memory  storage.  This  structure  (1) 
determines  the  nature  of  a  problem,  (2)  has  knowledge  of  the 
quality  of  one's  competence  with  respect  to  a  task,  ("5)  selects 
processes  for  solving  a  problem,  (4)  decides  on  a  strategy  for 
combining  these  processes,  (5)  decides  how  to  allocate  processing 
resources,  (6)  decides  how  to  represent  the  information  upon 
which  processes  act,  (7)  evaluates  how  well  cognitive  proceses 
are  accomplishing  subgoals  in  terms  of  overall  goals,  and  (8) 
makes  necessary  shifts  in  processing  strategies. 

This  structure  is  analogous  to  Sternberg's  metacomponent  and  to 
the  executive  processor  described  by  Snow,  Whitely,  and  others. 

The  LEXICAL  ACCESS/SEMANTIC-SYNTACTIC  ANALYSIS  STRUCTURE  is 
a  short  term  storage  or  working  memory  structure.  This  structure 
can  be  thought  of  as  the  workbench  where  information  is  held  for 
concentrated  cognitive  processing.  While  in  working  memory, 
information  is  highly  available  for  retrieveal.  It  can  be  held 
in  short  term  memory  while  it  is  being  operated  on  or  transformed 
by  cognitive  processes.  There  are  limitations  on  the  amount  of 
information  that  can  reside  in  working  memory.  One's  abilities 
to  process  such  information  are,  therefore,  limited. 

In  the  processing  of  text  materials,  a  match  must  be  made 
between  letter  strings  input  at  the  perceptual  processing  stage 
and  appropriate  semantic  referents.  Analysis  in  working  memory 
is  directed  at  attaching  meaning  to  perceptual  patterns.  For 
phrase  and  sentence  units,  analysis  is  also  directed  at 
organizing  these  meaning  elements  into  coherent  text 
representations.  Lexical,  semantic,  and  syntactic  knowledge  is 
called  upon  in  the  identification  of  words  and  in  phrase  and 
sentential  analysis. 

Lexical  Access  is  defined  simply  as  the  retrieval  of 


information  about  individual  words  from  long  term  memory.  In 
lexical  access,  grapheme  and  phoneme  data  drive  the  retrieval  of 
semantic  information. 

Semantic-syntactic  analysis  takes  place  in  short  term  memory; 
it  is  defined  by  the  retrieval  of  knowledge-based  structures  and 
information-free  functions.  These  structures  are  discussed  by 
Hunt  (1978).  In  semantic-syntactic  analysis,  the  knowledge-based 
and  information-free  long  term  memory  structures  are  accessed  and, 
in  the  case  of  the  information-free  functions,  executed  in  short 
term  memory  to  form  a  semantically  coherent  representation  of 
prose.  Information  about  individual  words  stored  in  long  term 
memory  is  retrieved  and  arranged  to  form  a  semantically  coherent 
structure.  Kintsch  and  van  Dijk  (1978)  have  developed  a  prose 
processing  model  which  references  the  types  of  knowledge-based 
structures  and  information-free  functions  involved  in  semantic- 
syntactic  analysis. 

The  fourth  structure  is  a  long  term  storage  structure.  This 
structure  is  the  locus  of  KNOWLEDGE-BASED  INFORMATION  STRUCTURES 
and  INFORMATION-FREE  FUNCTIONS.  The  knowledge-based  information 
structures  represent  semantic  and  syntactic  knowledge.  These 
structures  represent  the  ability  to  deal  with  words  and  the 
concepts  they  represent.  They  reflect  experience  with  and 
cognizance  of  the  English  language.  The  knowledge-based 
information  structures  are  also  associated  with  knowledge  of  the 
world  and  world  events.  These  knowledge  structures  are  mediated 
by  verbal  knowledge  but  represent  information  about  the  world 
ancillary  to  mastery  of  the  English  language. 

The  information-free  functions  are  the  operations  by  which 
information  structures  are  transformed  to  equivalent  structures 
necessary  for  task  performance.  No  semantic  or  syntactic 
information  is  associated  with  these  strategic  knowledge 
structures.  A  distinction  is  made,  here,  between  information  and 
processes.  These  operators  are  defined  by  learned,  stored 
transformation  rules.  Examples  of  information-free  operators  are 
the  processes  of  comparing  and  inferring.  These  operators 
perform  such  functions  as  identifying  similarities  and 
differences  among  information  structures,  generating  missing 
bridging  information  to  establish  semantic  coherence  for  a  text, 
or  sorting  information  structures  into  categories. 

The  final  structure  is  the  RESPONSE  OPERATOR.  This  is  the 
structure  through  which  appropriate  actions  are  selected  and 
executed.  The  response  operator  is  the  structure  by  which  the 
examinee  makes  either  an  observable  response,  such  as  selecting 
one  response  from  a  set  of  multiple  alternatives  or  makes  an  internal 
response  such  as  modifying  schema  in  long  term  memory. 


WK  and  PC  Item  Rating  Scheme 


Operational  definitions  were  developed  for  each  of  the 
operations/structures  described  above.  These  are  documented  in 
Mitchell  (1982).  The  operational  definitions  laid  the  foundation 
for  the  following  item  rating  scheme.  For  the  Word  Knowledge 
subtest,  items  were  coded  on  five  variables.  They  were: 

A.  The  total  word  count  for  target  string,  correct  response, 
and  incorrect  response  strings.  This  count  provided  an  index  of 
the  perceptual  processing  load  or  attentional  resource  allocation 
load  imposed  by  an  item.  It  was  hypothesized  that  increases  in 
item  difficulty  would  accompany  increases  in  the  encoding  or 
attentional  resource  load  presented  by  an  item. 

B.  Dichotomous  coding  for  the  lexical  Access/Semantic-Syntactic 
Analysis  variable.  Context-embedded  vocabulary  items  were  assigned 
a  unit  score.  Non-embedded  items  received  a  score  of  zero.  It  was 
proposed  that  decreases  in  item  difficulty  would  accompany  the 
provision  of  contextual  information  for  Word  Knowledge  items. 

C.  The  Kucera  and  Francis  word  frequency  score  for  the  target 
word.  This  value  indexed  the  probability  that  stored  information 
was  available  for  the  target  word.  It  was  hypothesized  that 
decreases  in  item  difficulty  would  accompany  increases  in  word 
frequency  scores  for  the  target  word. 

D.  The  Kucera  and  Francis  word  frequency  score  for  the  correct 
response.  This  value  indexed  the  probability  that  stored  semantic 
information  was  available  for  the  correct  response  string.  It  was 
hypothesized  that  decreases  in  item  difficulty  would  accompany 
increases  in  word  frequency  for  the  correct  response. 

E.  Count  of  the  high  frequency-correct  response/low  frequency- 
incorrect  response  pairs  for  each  item.  The  Kucera  and  Francis 
word  frequency  scores  were  used  for  the  correct  response  and 
incorrect  response  strings  to  generate  a  count  of  the  ’nigh-correct 
response/low-foil  response  pairs.  This  value  indexed  the  relative 
complexity  of  judgements  of  semantic  identity  for  correct 
response/target  word  and  incorrect  response/target  word  pairs.  It 
was  hypothesized  that  decreases  in  item  difficulty  would  accompany 
increases  in  the  number  of  high  frequency-correct  response/low 

f requency-incorrect  response  pairs. 

The  variables  for  the  PC  items  were: 

F.  The  total  word  count  for  the  paragraph,  correct  response, 
and  'incorrect  response  strings.  This  count  indexed  the  perceptual 
processing  or  attentional  resource  allocation  load  presented  by  an 
item.  It  was  hypothesized  that  increases  in  item  difficulty  would 
accompany  increases  in  the  encoding  or  attentional  resource  load 
presented  by  an  item. 
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G.  The  number  of  text-based  inferences  necessary  to  maintain 
semantic  coherence  as  defined  by  Kintsch  and  van  Dijk  (1978)  for 
the  target  paragraphs.  The  construction  of  text-based  inferences, 
a  controlled,  mechanistic  information-free  function,  calls  upon 
knowledge- based  information  structures.  A  description  of  the 
rating  system  appears  in  Turner  and  Greene  (1978).  It  was 
proposed  that  increases  in  item  difficulty  would  accompany 
increases  in  the  number  of  text-based  inferences  necessary  to 
maintain  semantic  coherence. 

H.  A  dichotomous  coding  for  the  response  inference  variable. 
Items  for  which  the  construction  of  an  inference  was  necessary  to 
link  the  correct  response  to  information  presented  in  the 
paragraph  selection  were  unit  coded.  Coding  followed  Turner  and 
Greene  (1978).  Again,  inference  construction  is  a  controlled, 
mechanistic  information-free  function  which  draws  upon  knowledge- 
based  information  structures.  It  was  hypothesized  that  item 
difficulty  would  be  positively  related  to  the  construction  of 
response  inferences. 

I.  The  number  of  propositions  per  paragraph  selection  divided 
by  the  number  of  words  in  the  selection.  Coding,  again,  followed 
Turner  and  Greene  (1978).  This  value  indexed  the  complexity  of 
propositional  structure  for  a  paragraph  selection.  It  was 
hypothesized  that  increases  in  item  difficulty  would  accompany 
increases  in  the  number  of  propositions  suggested  by  each 
paragraph  selection. 

J.  The  number  of  arguments  per  paragraph  selection  divided  by 
the  number  of  words  in  the  selection.  Argument  coding  agreed  with 
Turner  and  Greene  (1978).  Like  variable  four,  this  value  indexed 
complexity  by  processing  for  a  paragraph  selection.  It  was 
hypothesized  that  increases  in  item  difficulty  would  accompany 
increases  in  the  number  of  different  arguments  presented  by  each 
paragraph  selection. 

K.  The  mean  of  the  log  transformed  Kucera  and  Francis  word 
frequency  indices  for  the  arguments  of  each  selection.  The  value 
indexed  the  probability  that  stored  semantic  information  was 
available  for  the  arguments  of  a  paragraph  selection.  It  was 
hypothesized  that  decreases  in  item  difficulty  would  accompany 
increases  in  mean  word  frequency  scores  for  paragraph  selections. 

Obtaining  ratings  for  the  5  variables  on  the  70  Word 
Knowledge  items  and  the  6  variables  on  the  18  Paragraph  Comprehen¬ 
sion  items  took  approximately  80  rater  hours.  24  hours  were 
needed  to  obtain  a  second  set  of  ratings  on  variables  G-J  for  the 
18  Paragraph  Comprehension  items.  Percentage  agreement  rates  for 
codings  on  Paragraph  Comprehension  variables  G,  H,  I,  and  J  were 
61%,  11>%,  14%,  and  92^,  respectively. 


13  fiV 


Part  Two 


The  relative  effects  of  the  cognitive  processing  variables  on 
the  item  difficulty  parameters  for  the  verbal  items  of  the  ASVAB 
were  estimated  using  the  linear  logistic  latent  trait  model 
proposed  by  Fisher  11973)*  The  model  is  described  by  Whitely  and 
Schneider  (1981)  and  Thissen  (1982). 

The  model  is  a  one-parameter  latent  trait  model.  Item  diffi¬ 
culties  or  location  parameters  are  estimated  using:  (l)  item 
response  data  for  samples  of  examinees,  and  (2)  an  item  rating  ma¬ 
trix  reflecting  structural  characteristics  of  the  items.  In  these 
analyses,  the  item  rating  matrix  was  the  item-by-cognitive  component 
array  described  in  Part  One.  The  matrix  defines  a  set  of  linear  con¬ 
straints  on  the  estimation  of  the  item  location  parameters. 

The  one-parameter  latent  trait  model  estimates  item  difficulty 
in  the  following  way: 

'“ll"1’  i  ♦  «P(  t,  - 

«d*r«  C  j  -the  ability  level  for  person  J,  and 

e  i  "the  difficulty  for  ltm  1. 

The  linear  logistic  latent  trait  model  consists  of  a  set  of  linear 
constraints  on  item  difficulty  parameters,  such  that, 

•'i  ■  4  • 

where  tF;  is  a  vector  of  difficulties,  f;^  is  the 
item-by-cognitive  component  rating  matrix, is  a  vector  of 
component  parameters,  and  cj.  is  a  normalization  constant.  The 
component  parameters  reflect  the  relative  effects  of  the  cognitive 
component  variables  on  the  item  location  parameters. 

The  complete  linear  logistic  model  is  given: 


?  .  -  *XP<  llJ  "  ^  *  *?> 

(*1J  )  1  ♦  «p(  -  <£  flB  n*  *  «  >) 

When  the  -f||n  matrix  is  an  identity  matrix  and  x  is  set  to  0,  the 
linear  logistic  model  is  equivalent  to  the  one-parameter  latent 
trait  or  Rasch  model.  In  this  case,  each  item  defines  a  separate 
X*  item  difficulty  is  the  only  item  parameter. 

Part  Two  analyses  focused  on  eight  samples  of  examinees 
tested  on  ASVAB  8/9/10.  Six  of  the  samples  were  FY81  applicants; 
two  samples  were  FY81  Army  accessions.  Samples  one  and  two  were 
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Array  applicants  who  were  randomly  selected  without  regard  to 
whether  they  were  actually  admitted  to  the  Service.  Sample  one 
was  composed  of  2998  applicants  administered  form  one  of  the 
ASVAB.  Sample  two  had  2925  records;  these  applicants  took  form 
two. 


In  addition  to  these  two  samples,  subsaraples  of  applicants 
defined  by  level  of  verbal  ability  were  examined  for  each  form. 

Prom  the  two  samples  above,  the  twenty  percent  with  the  lowest 
Word  Knowledge  and  Paragraph  Comprehension  summed  score  were 
selected  and  the  twenty  percent  with  the  highest  Word  Knowledge  and 
Paragraph  Comprehension  summed  score  were  selected.  The  N's  for 
these  low  verbal  and  high  verbal  ability  samples  for  each  form  are 
at  Table  5. 


Table  5 

Part  Two  Applicant  Sample  Sizes 

ASVAB 


Verbal  Ability  Groups 


Form  1 

Form  2 

Heterogeneous  Ability 

2998 

2925 

Low  Ability 

585 

610 

High  Ability 

604 

609 

These  four  groups  were  subsaraples  of  applicant  samples  one  and  two. 

Two  samples  of  FY81  Array  accessions  were  also  selected.  The 
samples  of  accessions  were  students  enrolled  in  three  high-density 
Array  job  training  programs.  The  occupational  specialties  were: 

1 .  95B  Military  Police 

2.  94B  Pood  Service  Specialist 

3-  76C  &  76P  Equipment  Records  and  Parts  Specialist  and 
Material  Control  and  Accounting  Specialist 

Accession  sample  one  had  400  examinees;  sample  two  had  558  records. 


Rasch  item  difficulties  and  linear  logistic  latent  trait  item 
locations  were  estimated  for  applicant  and  accession  groups  using 


the  marginal  maximum  likelihood  procedures  outlined  by  Thissen 

(1982). 


For  the  linearly  constrained  latent  trait  model,  component 
parameters  were  also  calculated  for  each  form  and  sample.  The 
component  estimates  from  the  latter  analyses  were  examined,  and 
cognitive  component  variables  with  sizable  contributions  to  the 
estimation  of  item  difficulty  values  were  retained  in  a  reanalysis 
of  the  data.  For  each  group,  this  third  analysis  was  computed 
using  the  linear  logistic  latent  trait  algorithms.  A  reduced 
model  was  defined  by  the  third  analyses.  The  statistical  sig¬ 
nificance  of  the  individual  cognitive  component  parameters  was  not 
assessed;  estimation  of  standard  errors  for  the  component 
parameters  is  presently  computationally  intractable. 

The  adequacy  of  the  cognitive  component  variables  in 
predicting  difficulty  parameters  for  the  Word  Knowledge  and 
Paragraph  Comprehension  items  was  assessed  by  correlating  the 
Rasch  item  difficulties  with  the  linear  logistic  latent  trait 
difficulties  and  the  reduced  linear  logistic  latent  item 
locations.  The  Rasch  item  difficulties,  computed  with  one 
parameter  per  item,  functioned  as  criterion  scores  for  each 
sample.  The  two  sets  of  linearly  constrained  logistic  latent 
trait  item  locations  functioned  as  predictor  scores.  The  two 
simple  correlations  indexed  the  strength  of  the  relation  between 
the  unconstrained  and  contrained  item  difficulties  estimated  in 
each  applicant  and  accession  group. 


RESULTS 

Development  of  the  Word  Knowledge  Models 

The  full  and  reduced  models  for  the  Word  Knowledge  subtests 
are  at  Tables  6-9*  The  correlations  between  the  Rasch  item 
difficulties  and  the  five-variable  linearly-constrained  item 
difficulty  values  for  the  Word  Knowledge  subtests  ranged  from  .45 
to  .55*  There  were  essentially  no  differences  in  the  correlations 
across  forms  or  subjects. 

Reduced  linear  logistic  latent  trait  models  were  developed 
for  each  Word  Knowledge  form  and  sample.  Three  sets  of  values 
were  considered  in  the  selection  of  cognitive  component  variables 
for  the  reduced  models.  They  were:  (l)  the  cognitive  component 
parameters  estimated  by  the  linear  logistic  latent  trait  algo¬ 
rithms,  (2)  the  intercorrelations  between  items  ratings  on  the 
five  Word  Knowledge  variables,  and  (5)  the  simple  correlations 
between  the  item  ratings  and  the  Rasch  or  unconstrained  item 
difficulty  values. 
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Pull  Model  Cognitive  Component  Parameters  (df=3*t) 


Table 
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Table  9 

luced  Model  Cognitive  Component  Parameters  (df=3*0 
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Cognitive  Component  Variables 


The  cognitive  component  parameters  for  both  the  Total  Word 
Count  and  Lexical  Access/Semantic-Syntactic  Analysis  variables 
were  sizable  for  all  examinee  groups  and  both  forms.  These 
variables  were  entered  singly  in  the  reduced  models  because  item 
ratings  for  the  two  variables  were  highly  correlated.  The  word 
count  for  an  item  was  positively  related  to  whether  the  target 
word  was  content  embedded  or  non-embedded .  The  Total  Word  Count 
variable  was  selected  for  the  hetereogeneous  and  low  verbal 
ability  applicants  and  for  the  accessions  taking  form  one.  The 
Lexical  Access/Semantic-Syntactic  Analysis  variable  was  included 
for  the  high  ability  applicants  taking  form  one  and  for  all 
examinees  administered  form  two. 

The  sign  of  the  cognitive  component  parameter  for  the  Total 
Word  count  variable  in  the  form  one  reduced  models  was  positive. 

The  cognitive  component  parameters  for  the  Lexical  Access/ 
Semantic-Syntactic  Analysis  variable  for  form  two  analyses  were 
negative.  The  direction  of  effect  was  consistent  with 
hypothesized  relations  for  both  variables.  Item  difficulty  was 
positively  related  to  the  total  number  of  words  presented  by  an 
item.  Item  difficulty  decreased  with  the  provision  of  contextual 
information  for  the  Word  Knowledge  items. 

There  was  evidence  of  interacting  effects  for  the  Total  Word 
Count  and  Lexical  Access/Seraantic  Syntactic  Analysis  variables 
when  they  were  entered  simultaneously  in  the  five-variable  models. 

A  possible  interpretation  of  the  interaction  is  that  item  dif¬ 
ficulty  does  not  increase  with  heavy  encoding  requirements  when 
the  additional  lexical  units  provide  context.  Because  the  decre¬ 
ments  in  the  item  difficulty  correlations  from  the  five-variable 
linear  logistic  latent  trait  models  to  the  reduced  models  were 
minimal  when  the  variables  were  entered  singly,  these  two  vari¬ 
ables  were  not  combined  in  the  final  models. 

The  cognitive  component  parameters  for  the  Target  Word 
Frequency  variable  for  form  one  and  the  Correct  Response  Word 
Frequency  variable  for  form  two  were  sizable.  The  Correct 
Response  Word  Frequency  component  parameter  for  the  low  verbal 
ability  group  taking  form  one  was  also  large.  The  Target  Word 
Frequency  variable  was  entered  in  the  reduced  models  for  all 
groups  taking  form  one.  The  Correct  Response  Word  Frequency 
variable  was  included  for  the  low  verbal  ability  group  taking  form 
one  and  for  all  examinees  taking  form  two. 

Like  the  Total  Word  Count  and  Lexical  Access/Seraantic-Syntac- 
tic  Analysis  variables,  item  ratings  on  these  two  variables  were 
highly  correlated.  Difficult  target  words  were  accompanied  by 
difficult  defining  words.  The  direction  of  effect  was  consistent 
with  hypothesized  relations  for  these  two  variables  in  the  reduced 
models.  Words  with  low  frequency  counts  were  positively  related  to 


item  difficulty  both  for  the  target  word  and  the  correct  alterna¬ 
tive. 

The  cognitive  component  parameters  for  the  High  Frequency 
Correct  Response/Low  Frequency  Incorrect  Response  variable  were 
small  for  all  groups.  This  variable  was  operationalized  in  such  a 
way  that  its  values  were  highly  related  to  the  values  assigned  on 
the  Correct  Response  Word  Frequency  variable. 

The  parameter  estimates  for  the  cognitive  component  variables 
could  not  be  evaluated  for  significance.  In  addition,  the  prob¬ 
abilities  would  be  meaningless  in  light  of  the  model-building 
procedures  used.  Only  the  relative  effects  of  the  cognitive 
processing  variables  included  in  the  models  could  be  examined. 

The  correlations  between  the  reduced  model  item  difficulty 
estimates  and  the  Rasch  item  difficulty  values  ranged  from  .41  and 
•54*  The  correlations  were  quite  similar  to  the  correlations  for 
the  five-variable  models.  The  models  explained  from  17^  to  30^  of 
the  variance  in  the  item  difficulty  values  estimated  by  the  Rasch 
models. 


Form  Differences 

The  reduced  models  for  the  two  forms  of  the  Word  Knowledge 
subtest  were  judged  to  be  conceptually  similar.  Both  sets  of 
models  contained  a  variable  which  indexed  the  amount  of  informa¬ 
tion  presented  in  the  item  and  both  included  a  variable  which 
indexed  the  difficulty  of  the  target  or  response  words. 

The  Total  Word  Count  variable  is  a  measure  of  the  amount  of 
information  that  must  be  encoded,  analyzed,  and  retained  by  the 
reader.  The  Lexical  Access/Seraantic-Syntactic  Analysis  variable 
indexes  the  provision  of  contextual  information  for  the  target 
word.  For  the  Target  Word  Frequency  and  Correct  Response  Word 
Frequency  variables,  item  ratings  reflect  the  probability  that 
examinees  have  encountered  and  comprehended  information  about 
individual  lexical  units.  For  both  forms  item  ratings  for  the 
Total  Word  Count  and  Lexical  Access/Semantic  Syntactic  Analysis 
variables  and  the  Target  Word  Frequency  and  Correct  Response  Word 
Frequency  variables  were  highly  correlated. 


Group  Differences 

Although  differences  in  estimated  Rasch  and  linear  logistic 
latent  trait  item  difficulty  values  were  observed  for  examinees 
differentiated  by  level  of  verbal  ability,  no  marked  differences 
between  the  cognitive  component  parameters  were  noted  for  groups 
of  examinees  on  the  two  forms.  The  composition  of  the  models  and 
relative  magnitude  of  the  cognitive  component  parameters  were 


comparable  for  hetereogeneous ,  low,  and  high  verbal  ability  FY81 
applicants  and  FY81  accessions. 


Development  of  the  Paragraph  Comprehension  Models 

The  results  of  analyses  on  the  Paragraph  Comprehension 
subtests  were  difficult  to  interpret.  Analyses  were  performed  on 
nine  items  for  each  subtest  and  relations  were  hypothesized  for 
six  cognitive  processing  variables.  The  ratios  of  the  number  of 
parameters  to  the  number  of  items  for  both  the  full  and  reduced 
models  were  such  that  meaningful  interpretation  was  prohibited. 

The  results  below  are  presented  merely  for  completeness.  The  full 
and  reduced  models  appear  at  Tables  10-13* 

Cognitive  component  variables  were  selected  in  the  same  way 
for  the  reduced  Paragraph  Comprhension  models  as  they  were  for  the 
reduced  Word  Knowledge  models.  The  cognitive  component  parameters 
for  the  #  Propositions/#  Words,  #  Arguments/#  Words,  and  Mean 
Argument  Frequency  variables  for  form  one  were  high  for  the 
hetereogeneous  ability  level  applicants,  the  high  ability  ap¬ 
plicants,  and  the  accessions.  For  the  low  ability  group,  the 
absolute  values  of  the  Word  Count,  Response  Inference,  #  Proposi¬ 
tions/#  Words,  and  Mean  Argument  Frequency  component  parameters 
were  high. 

The  correlations  between  the  six-variable  model  item  dif¬ 
ficulties  and  the  Rasch  item  difficulty  estimates  ranged  from  .81 
to  .93  for  Form  One.  Correlations  of  this  magnitude  were  presup¬ 
posed  by  the  fact  that  the  number  of  parameters  was  only  slightly 
less  than  the  number  of  items.  For  the  reduced  models,  the 
correlations  ranged  from  .67  to  .84.  There  were  no  discernable 
differences  between  groups. 

For  Form  Two  the  correlations  between  the  linear  logistic 
latent  trait  difficulty  estimates  and  the  Rasch  item  difficulties 
ranged  from  .86  to  .99.  The  cognitive  component  parameters  for 
the  Response  Inference,  #  Propositions/#  Words,  and  #  Arguments/# 
Words  variables  were  high  for  the  hetereogeneous  applicant  group, 
the  low  ability  group,  and  the  accession  group.  The  Response 
Inference  variable  did  not  have  a  sizable  weight  for  the  high 
ability  group. 

The  correlations  between  the  linear  logistic  latent  trait 
difficulty  values  for  the  reduced  models  and  the  Rasch  dif¬ 
ficulties  ranged  from  .81  to  .96.  These  models  represented  from 
66#  to  92#  of  the  variance  in  the  item  difficulties  estimated  by 
the  Rasch  models.  There  appeared  to  be  no  group  differences. 
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FUll  Model  Cognitive  Component  Parameters  (df=8) 
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Table  12 

pull  Model  Cognitive  Component  Parameters  (df=8) 
Paragraph  Comprehension,  Form  Two 
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DISCUSSION 


Relevance  of  Cognitive  Processing  Variables  to  Item  Difficulty 

Part  2  analyses  suggested  that  cognitive  processing 
variables  were  related  to  item  difficulty  for  FY81  Army 
applicants  and  accessions  on  the  verbal  subtests  of  the 
ASVAB.  The  reduced  linear  logistic  latent  trait  models 
predicted  from  17%  to  30f  of  the  variance  in  the  item 
difficulty  values  estimated  by  the  Rasch  models  for  the 
Word  Knowledge  items.  The  reduced  models  were  predictive 
of  the  Rasch-computed  item  difficulties  for  the  Paragraph 
Comprehension  items. 

Additional  information  about  the  relations  between 
cognitive  processing  requirements  and  item  difficulty  might 
be  gleaned  in  future  research  using  item  sets  specifically 
developed  to  represent  a  varied  range  of  proficiency  on  the 
individual  processing  variables.  More  sensitive  assessment 
of  individual  differences  on  these  variables  and  of  their 
relevance  to  item  difficulty  might  be  attained  with 
instruments  constructed  with  special  attention  to  assessing 
a  varied  range  of  ability  levels  on  the  processing 
variables.  Evaluation,  here,  was  possible  only  on  the 
cognitive  constructs  represented  on  the  operational  ASVAB 
and  only  in  terms  of  the  component  variance  built  into  the 
test. 


Further  information  about  the  relations  between 
cognitive  processing  variables  and  item  difficulty  might 
also  be  gained  by  fitting  multidimensional  item  response 
models  to  these  data.  The  present  analyses  fit 
unidimensional  item  response  models  to  data  which  were 
multidimensional.  Both  theoretical  and  empirical 
considerations  attest  to  the  multidimensionality  of  the 
data. 


Determinations  of  the  Parrallism  of  the  Operational  Forms 

( 

There  was  no  evidence  to  suggest  that  the  two 
operational  forms  of  the  ASVAB  under  investigation  were  not 
parallel.  There  were  essentially  no  differences  in  item 
ratings  for  the  five  Word  Knowledge  and  six  Paragraph 
Comprehension  variables  across  the  two  forms.  The 
cognitive  component  examinations  of  the  two  forms  did  not 
point  clearly  to  differences  between  forms. 
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CONCLUSION 


The  relations  of  cognitive  processing  variables  to  item 
difficulty  for  the  ASVAB  8/9/10  Word  Knowledge  and 
Paragraph  Comprehension  subtests  were  demonstrated. 
Assessments  of  the  predictive  utility  of  these  and  similar 
constructs  to  success  in  Army  training  and  military  jobs 
are  needed. 

It  is  proposed  that  analyses  of  this  type  allow  for  the 
assessment  and  documentation  of  the  contributions  of 
cognitive  processing  operations,  stores,  and  strategies  to 
performance  on  industrial/organizational  assessment 
instruments.  These  techniques  can  inform  evaluations  of 
the  predictiveness  and  relevance  of  aptitude  and  ability 
measures.  These  methods  can  be  used  to  develop  item  sets 
so  that  specified,  training-  and  job-relevant  cognitive 
processing  abilities  are  tapped. 
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